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Executive Summary 


Background 


Following the release of A Framework for K-12 Science Education (National Research 
Council, 2012) and anticipating the release of the Next Generation Science Standards 
(NGSS; NGSS Lead States, 2013), the National Research Council [NRC] released a 
report entitled Monitoring Progress Toward Successful K-12 STEM Education (NRC, 
2013). This Monitoring Progress report contained recommendations to the National 
Science Foundation on how to evaluate the status of science, technology, engineering, 
and math (STEM) education in the US. The report recommended 14 indicators that 
could be used to drive and track improvement in STEM education across the nation. 


Recognizing the need to assess the status of STEM education, the National Science 
Foundation invited proposals for research projects focused on the indicators 
recommended by the Monitoring Progress report. This report is a product of one of the 
resulting projects. In this project, BSCS set out to address Indicator 4 from Monitoring 
Progress, which called for analyzing the degree to which the most widely used science 
and mathematics materials in US schools promote the vision of education described in 
the Common Core standards for math and the NRC Framework (NRC, 2013, p. 19). We 
interpreted the task of analyzing instructional materials as consisting of two 
components: (1) assessing the degree of alignment of instructional materials with the 
vision and goals of the NRC Framework and NGSS and (2) assessing the likelihood of 
the materials supporting teachers and students in bringing the vision and goals to life. 


In approaching this task, we built on considerable work done over the last two decades 
on defining and measuring the quality of instructional materials (e.g., BSCS, 2007; 
Davis & Krajcik, 2005; Duschl, Schweingruber, & Shouse, 2007; Edelson, 2001; 
Kesidou & Roseman, 2002; Roseman, Stern, & Koppal, 2010). However, the vision of 
science teaching and learning presented by the NRC Framework and the NGSS has 
new elements that have not been part of previous assessments of instructional 
materials, for example, the explicit inclusion of science and engineering practices 
(SEPs), crosscutting concepts (CCCs), and disciplinary core ideas (DCIs). Therefore, 
an important part of the work on this project has been to identify and explain these new 
components to support assessment of them in instructional materials. 


Overall, the Guidelines for Evaluating Science Instructional Materials (hereafter, the 
Guidelines) provide answers to the following questions: 


e What criteria should be used to evaluate the extent to which instructional 
materials that support the vision of science education described in the NRC 
Framework and NGSS? 

e What tools and processes should be used to evaluate instructional materials 
based on these criteria? 
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About the Guidelines 


To conduct the review of instructional materials called for by the Monitoring Progress 
report will ultimately require the development of a fully specified set of tools and 
processes for evaluating instructional materials. Developing this complete set of tools 
and processes was beyond the scope of this project. The project’s objective was to 
create guidelines for creating those tools and processes. Specifically, the Guidelines 
identify criteria for use in evaluating instructional materials as well as characteristics of 
tools and processes for applying those criteria. These guidelines are designed to 
provide a blueprint for the work of creating tools and a process that can be used to 
conduct valid and reliable assessments of instructional materials. 


Having developed these guidelines for the goal of evaluating the most widely used 
materials across the country as recommended by the NRC Monitoring Progress report, 
we are also mindful of the fact that there are many other reasons that people conduct 
reviews and evaluations of instructional materials—such as textbook adoption 
decisions, teacher professional development, and revisions to materials to enhance 
their quality and alignment with the NGSS. We anticipate that these guidelines will have 
some value for the design of tools and processes for these purposes, as well. 


The Guidelines report was developed over two years through a process that involved 10 
BSCS researchers and 18 leaders in science education from across the country. The 
team brought to the project expertise in (1) the NRC Framework and the NGSS, (2) the 
development and use of tools and processes to evaluate the quality of science 
instructional materials, (3) curriculum development, and (4) policy making. The 
multistage development process included a synthesis of relevant literature; a convening 
of science education leaders to set the basic parameters for the Guidelines; the drafting 
of the evaluative criteria and guidelines for tools and processes; and multiple rounds of 
review, feedback, and revision. 


The evaluative criteria for materials are summarized in Table 1. They include criteria for 
evaluating both student and teacher materials across four categories: learning goals, 
coherence, learning experiences, and monitoring learning. The guidelines for tools and 
processes are summarized in Table 2. 
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Table 1. Criteria for evaluation of instructional materials in science. 


When evaluating student materials, consider the extent 


to which ... 


When evaluating teacher materials, consider the 
extent to which ... 





NGSS-Driven Learning Goals 





1S Materials are based on learning goals. Those goals 
call for learning of 


e disciplinary core ideas, science and 
engineering practices, crosscutting concepts 
from NGSS integrated as three-dimensional 
learning; 


e the nature of science, engineering, technology, 
and applications of science from NGSS; and 


e Common Core State Standards for English 
language arts and mathematics. 


1T Materials explain the learning goals; the rationale 
for selecting them; and 


e how they promote three-dimensional learning; 
e how they promote learning of the nature of 


science, engineering, technology, and 
applications of science; and 


e how they promote learning of the Common 
Core standards for English language arts and 
mathematics. 





2S Materials use phenomena or problems to focus 
students on learning goals. 


2T Materials explain how the phenomena or 
problems are used to focus students on learning 
goals. 





3S Materials are based on scientifically accurate and 
grade-level-appropriate learning goals. 


3T Materials situate learning goals within the 
progression of K-12 learning laid out by the 
NGSS. 





Coherence Across Three Dimensions 





4S Materials are designed with carefully sequenced 
learning goals and well-matched experiences. 


4T Materials communicate the design principles and 
sequencing underpinning the storyline. 





5S Materials provide students with opportunities to 
make links across the three dimensions to build 
coherent conceptual understanding and abilities to 
use the practices. 


5T Materials promote teacher knowledge-building 
related to the storyline. 





Learning Experiences Across Three Dimensions 





6S Materials provide multiple opportunities for 
students to share and negotiate their ideas, prior 
knowledge, and experiences. 


6T Materials support teachers in anticipating common 
student ideas and include guidance to elicit and 
challenge student thinking. 





7S Materials use motivating contexts to engage 
students in real-world phenomena and authentic 
design problems. 


7T Materials provide guidance to teachers for using 
effective teaching strategies that engage students 
in real-world phenomena and authentic design 
problems. 





8S Materials are accessible to a wide range of 
students. 


8T Materials provide suggestions for how to address 
a range of students’ skills, needs, and interests. 





Monitoring Learning Across Three Dimensions 





9S Materials include accessible and unbiased 
formative and summative assessments of students’ 
three-dimensional learning. 


9T Materials highlight formative and summative 
assessments and provide tools and guidance for 
interpreting evidence of three-dimensional 
learning and using assessment results to plan for 
future instruction. 





10SMaterials include multiple opportunities for self- 
assessment and reflection to promote 
sensemaking among students. 


10T Materials provide guidance for teachers to use 
data from assessments to provide feedback to 
students and promote student self-assessment 
and reflection. 
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Table 2. Guidelines for Assessing the Quality of Instructional Materials 
The evaluation system 
includes both tools and processes. 
includes a guide for evaluators. 
specifies a summary report that justifies the evaluation results and offers suggestions for 
modifying instructional materials to enhance their quality. 


The evaluation system should be supported by | The evaluation system should include Processes 
Tools that that 


4. specify what to look for as evidence for each | 7. identify appropriate units of analysis. 
Evaluative Criterion. 


5. have clearly defined scoring guidelines for 8. involve dialogue and consensus-building 
capturing evidence from materials. among a team of evaluators. 


6. include forms for documenting specific 9. assure consistency across evaluators. 
evidence of the Evaluative Criteria and 
suggestions for improvement. 
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